Endoscope and imaging arrangement providing improved depth of field and resolution

ABSTRACT

A dynamic imaging system of an endoscope that adjusts path length differences or focal points to expand a usable depth of field. The imaging system utilizes a variable lens to adjust the focal plane of the beam or an actuated sensor to adjust the detected focal plane. The imaging system is thus capable of capturing and adjusting the focal plane of separate images captured on separate sensors. The separate light beams may be differently polarized by a variable wave plate or a polarized beam splitter to allow separate manipulation of the beams for addition of more frames. These differently focused frames are then combined using image fusion techniques.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part of U.S. application Ser. No.15/175,520, filed on Jun. 7, 2016, and entitled “Endoscope and imagingarrangement providing depth of field,” that is hereby incorporated byreference.

BACKGROUND OF THE INVENTION Field of the Invention

The presently disclosed device is directed to an endoscope capable ofproducing views with increased depth of field. The endoscope can utilizea variety of beam splitters combined with polarization filters andbeam-shapers to provide separate imaging channels to separate sensors.

Description of the Background Art

Conventionally, endoscopes were monocular providing images through longtelescoping lens arrangements. Initially, they allowed doctors to viewinside patients with their eyes. These simple devices relayed images butdid not provide depth information. As video-assisted surgery progressed,depth and size information was increasingly necessary, particularly, fortumor and thrombosis identification and assessment.

The monocular endoscopes were modified to take in two views at the headand switch between each view, transmitting each one across aconventional single channel lens to a single sensor. For instance, thedevice described in U.S. Pat. No. 5,944,655 to Becker is exemplary.These devices provided stereoscopic views for doctors but requiredthicker heads to accommodate the separate imaging capturing lenses, beamcombiners and high-speed apertures. This made smaller scale applicationsdifficult to produce.

Alternatively, some devices provided two separate channels for each viewand separate sensors in a mirrored configuration much like twoside-by-side monocular endoscopes. This arrangement conserves headvolume but at the expense of a thicker channel between the head and thesensor. The device disclosed in US 2014/085421 is exemplary of the stateof the art. These two separate channels still only provide stereoscopicviews; not three-dimensional information or depth of field information.

Another endoscope arrangement is disclosed in US 2014/0198194. Thisarrangement uses only a single image formation and transmittal channel,but splits the channel at the image sensing end. FIG. 1 of US2014/0198194 is illustrated in FIG. 1 of this disclosure. The beamsplitter 5 at the distal end of the endoscope 1 divides the incomingcaptured light into two channels. The first channel is reflected upwardsby the first interface of the prism 2 and polarized by the λ/4 waveplate4 before being reflected by mirror 3 back to the sensor 7. The secondchannel passes through the interface and through the second prism 6 to asecond interface and is reflected down to the sensor 7.

Endoscope 1 of US 2014/0198194 also provides two views from the twoimaging channels. Each channel is separately focused due to path lengthdifferences within the prism. These separate channels allow for a depthof field to be reconstructed from the two separately focused images.However, the depth of field difference provided by the device of FIG. 1is static. Thus, depth information can only be provided at two focalplanes. This limits the amount of in-focus image information which canbe utilized from the two images.

SUMMARY OF THE INVENTION

The invention disclosed herein relates to a dynamic imaging system foradjusting path length differences to expand a usable depth of field foran endoscope. The imaging system utilizes a variable lens to adjust thefocal plane of the beam or an actuated sensor to adjust the detectedfocal plane. The imaging system is thus capable of capturing andadjusting the focal plane of separate images captured on separatesensors. The separate light beams may be differently polarized by avariable wave plate or a polarized beam splitter to allow separatemanipulation of the beams.

The imaging system can be designed for easy attachment to an endoscope.The optics can be adapted to receive images at a number of sensors bysplitting the incoming beam. Different beam splitter configurations areprovided to deliver two or more beams of different path lengths todifferent sensors. These captured images with different focal planesrequire additional processing to generate a combined image.

The image fusion methods of Mertens, et al. “Exposure Fusion” byMertens, et al. in Computer Graphics and Applications (2007) and Burt,et al. “A Multiresolution Spline With Application to Image Mosaics” ACMTransactions on Graphics, Vol. 2. No. 4, October 1983, p. 217-236 areadapted to combine the differently focused images of the imaging systeminto one clearer image. The combination of these processes can handlefocal variations (far and near) as well as exposure differences (overand under). First the fusion method generates a contrast weight map, asaturation weight map and an exposure weight map for each capturedimage. Second, these maps are applied to select the best pixels fromeach image. Finally, the separate weighted images containing theselected or weighted pixels are combined with pyramid-based imagefusion. The journal article “Exposure Fusion” by Mertens, et al. inComputer Graphics and Applications (2007) is incorporated herein byreference. Likewise, Burt, et al. “A Multiresolution Spline WithApplication to Image Mosaics” ACM Transactions on Graphics, Vol. 2. No.4, October 1983, p. 217-236 is incorporated herein by reference.

The imaging system is placed in an adaptable camera head for anendoscope, such that the camera head can be placed on a variety ofendoscopes. In addition to the beamsplitting and polarizing optics, thecamera head would include Radio Frequency Identification receiver fordetecting the endoscope end and aiding in the coupling procedure. Upondetection of the particular endoscope being used, the imaging systemwould adapt the sensor positions and other optical elements as necessaryto use the light beam from the particular endoscope.

Further scope of applicability of the present invention will becomeapparent from the detailed description given hereinafter. However, itshould be understood that the detailed description and specificexamples, while indicating preferred embodiments of the invention, aregiven by way of illustration only, since various changes, combinations,and modifications within the spirit and scope of the invention willbecome apparent to those skilled in the art from this detaileddescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become more fully understood from thedetailed description given hereinbelow and the accompanying drawingswhich are given by way of illustration only, and thus are not limitativeof the present invention, and wherein:

FIG. 1 shows a known endoscope lens arrangement;

FIG. 2 shows a first beam splitter and imaging arrangement;

FIG. 3 shows a second beam splitter and imaging arrangement;

FIG. 4 shows an imaging head of an endoscope;

FIG. 5 shows two separate imaging channels to one of the sensors of theimaging head of FIG. 4;

FIG. 6 shows two additional imaging channels to the other sensor of theimaging head of FIG. 4;

FIG. 7 shows an alternatively arranged imaging head of an endoscope;

FIG. 8 shows an imaging head with a moveable lens for an endoscope;

FIG. 9 shows an alternatively arranged imaging head of an endoscope;

FIG. 10 shows an endoscope imaging head and a connection to an insertionend of the endoscope;

FIG. 11 shows an alternatively arranged imaging head of an endoscope;

FIG. 12 shows a segment of an alternatively arranged endoscope;

FIG. 13 shows several segments in a series arrangement;

FIG. 14 shows an aperture; and

FIG. 15 shows an imaging head for connection to the insertion head.

DETAILED DESCRIPTION

The beam splitter of FIG. 2 receives a beam of light from a carrier lens12. The beam enters the pentaprism 8 and is partially reflected off afirst interface 9 and then internally reflected back towards a sensor11. The remaining light passing through the interface is reflected off aback surface of a prism 10 towards the same sensor 11. The firstinterface 9 can be a half-silvered mirror or semi-reflective coating onone of the prism surfaces.

Each reflection changes the path length and, as a result, the back focallength of each beam is different. The image formed on each portion ofthe sensor 11 captures a separate focal plane of the object beingobserved by the insertion portion of an endoscope. Alternatively, twoseparate sensors can be used in place of the single sensor 11, and theindividual sensors can be placed at different distances from the beamsplitter.

The beam splitter of FIG. 3 includes the beam splitter of FIG. 2 alongwith prisms 8 and 10 with an additional rear prism 14. This arrangementfurther splits the remaining light passing through the interface 15. Theinterface 15 can be ⅔ reflective or 50% reflective and the sensors 11,individually detecting each partial beam, can be offset from the prismsat different distances. The additional interface 15 provides a thirdbeam which is captured by a third sensor 11 for additional depth offield information.

Each separate sensor 11 detects a differently focused beam providing animage including information at a particular depth. Each of the beamsplitter elements or prisms (8, 10 and 14), can be made of crystalglass, polymers, acrylic, or other light transmissive materials. Also,the interfaces (9, 15) can be made partially reflective such that theintensity of each sensed beam is substantially equal. Alternatively, tocompensate for surface losses or other differences, the reflectivesurfaces on the interfaces (9,15) may divide the beams unequally.

The optical arrangement of FIG. 4 describes an imaging head for anendoscope, including polarizing beam splitter 16 with a polarizingfunction, variable liquid lens 20, temporally-switched variablewaveplate 17, beam splitter 19 and sensors 18. The first beam splitter16 separates the light exiting a detachable endoscope into two channels.A focus difference is introduced between the two channels while they areseparate using the variable liquid lens 20. The two channels are thenrecombined spatially while maintaining their distinct polarizations.

Both channels pass through a gap in which the variable liquid lens 20 isdisposed. After passing through this gap, both channels enter anotherbeam splitter 16 recombines the two beams and passes them on to avariable wave plate 17 for changing the polarization of both beams. Thevariable wave plate 17 varies between ON and OFF such that when it isON, the polarization of the incoming light beam is rotated by 90degrees.

After the variable wave plate 17, the combined beam enters a beamsplitter 19 which once again separates the channels based onpolarization such that they are imaged onto different sensors 18. Thus,on odd frames, one sensor 18 captures “s” polarized light and the othersensor 18 captures “p” polarized light. On even frames, the differentsensors 18 are given the other channel. The collimating lens group 22 isdisposed before the variable wave plate 17 for further beam manipulationand control.

In this manner, four different images corresponding to four differentfocal planes can be acquired over the course of two frames. From theplurality of images, a processing unit (not shown) calculates an imagewith greater depth of field than would be possible with a single sensorand conventional optics. Alternatively, this imaging head could beintegrated into a videoendoscope that does not detach from a camerahead.

The two illustrations in FIG. 5 show the two beam paths taken by thefirst channel during separate capture frames, that is on odd/evenframes. Likewise, the two illustrations in FIG. 6 show the two beampaths taken by the second channel during separate capture frames. Thebeams in FIG. 5 have different focal planes due to a difference inoffset of the sensors 18 from the beam splitter 19. Likewise, beams inFIG. 6 have different focal planes due to a difference in offset of thesensors 18 from the beam splitter 19.

The focal difference between the first and second channels due to thevariable lens 20 and path length difference is also simultaneouslyprovided to the sensors 18. This results in four unique focal planesover two capture periods. Furthermore, the variable lens can changeposition or focal power to increase the number of focal planes furtheror simply to adjust focus. The variable lens may have variable curvatureor a variable index of refraction.

The camera head may also have a control unit to adjust the focaldifference according to the content of the acquired image, the imagingenvironment, or other application specific needs. In addition, thecamera head can be capable of identifying the specific endoscope orendoscope-type being used and adjust focus accordingly. The “s” and “p”polarization described above is exemplary and could be replaced withcircular or elliptical polarization.

The camera head of FIG. 7 includes a single beam splitter 16, a variablelens 20, two collimating lens groups 22, and two sensors 18. Thefunction of the camera head is similar to that of FIGS. 4-6 with norecombination of the two channels by an additional beam splitter 16.This configuration is easier to manufacture but only acquires two imagesper acquisition period.

In the arrangement of FIG. 7, the light beam from the endoscope entersthe beam splitter 16 and is divided into two beams. The first beampasses through a collimating lens group 22 before impinging on the imagesensor 18. Separately, the second beam passes through a variable lens 20and then another separate collimating lens group 22 before impinging onimage sensor 18.

The arrangement is also connected to a control device for controllingthe variable lens and a processor 71 that calculates depth from thecaptured images or segments the in-focus portions for recombination anddisplay. The processor 71 is also able to model three-dimensionalsurfaces and build complex tissue models. These models and surfaces canbe stored in memory such as RAM or transmitted to a display screen fordisplay to a user.

Conventional methods to increase depth of field fundamentally decreasethe resolution. Thus, typically systems are forced to make a tradeoffbetween depth of field and resolution. However, combining several imagesensors to provide depth information preserves resolution and can evenimprove it. Furthermore, the images can be segmented to provide thein-focus portion of each captured image and recombine the in-focussegments for a clearer image with more depth information and depth offield.

Additionally, the camera head of FIG. 7 can include an aperture stopsmaller than the exit pupil of the endoscope. This increases the depthof field but also reduces the resolution. Alternatively, the camera headcould provide a fixed offset between focal planes. However, to provide afixed offset across different endoscopes, the camera head would need toadjust to each endoscope type. The camera head can be integrated into ordetachable from a video-endoscope.

The camera head can also be simplified by replacing the variable liquidlens 20 with a simple movable focusing lens 23 as shown in FIG. 8. Thismovable lens 23 can change the focus position or adjust the focal planebut cannot vary in focal power as can variable liquid lens 20.Otherwise, the system of FIG. 8 is substantially the same as that ofFIG. 7 with similar capabilities.

Another optical arrangement for providing depth of field, as in theprevious arrangements, is shown in FIG. 9. In this configuration, thevariable liquid lens 20 are placed after a single beam splitter 19.Thus, the light beam received from the endoscope first passes through acollimating lens group 22 and a variable wave plate 17 then into thebeam splitter 19 where the light beam is divided into two beams. Thefirst beam passes straight through the beam splitter and a spectralfilter 21 to the image sensor 18 and is captured. The second beam isdeflected at a right angle and passes through the variable liquid lens20 where the focal plane of the beam is changed. The second beam is thenreceived and captured by a separate image sensor 18.

The arrangement in FIG. 9 is capable of imaging two or more images ofdifferent focal planes per acquisition period. Additional focal planescan be acquired subsequently by adjusting the variable liquid lens 20between each image capture. Furthermore, the position of each imagesensor 18 of FIG. 7-9 can be adjusted based on the endoscope beingattached, environmental variables or object distance.

The camera head can identify the endoscope being attached and store inmemory or adjust automatically based on a detection of a specificendoscope type, where the variable liquid lens 20 or the relativepositions of the sensors 18 are adjusted. In either case, the adjustmentpreferably optimizes the focal offset introduced by these elements.Furthermore, the ray bundles at the focal planes should be telecentric.

The larger system diagram of FIG. 10 illustrates how any of the cameraheads disclosed herein would interact with an endoscope optical head 24.In the illustrated case, the variable liquid lens 20 is placed after thebeam splitter 19. In addition, two collimating lens groups 22 are placeddownstream of the beam splitter 19. The rod lens endoscope 24 could beany conventional endoscope or one specially adapted to identify with thedisclosed camera head. In addition, a collimating lens group 22 isdisposed upstream from the beam splitter 19 to control and adapt thelight beam received from the endoscope.

An alternative arrangement without variable liquid lenses 20 is providedin FIG. 11 for a simpler structure and cheaper camera head of anendoscope. The incoming light beam is split into two beams by a beamsplitter 19. Each beam then passes through an aperture stop 13, eachaperture stop 13 having a different diameter. The difference indiameters produces images with distinct depth of field and resolutioncharacteristics. Each beam then passes through carrier lenses beforebeing collimated and reflected by the respective mirrors 25 onto thesensors 18. Lenses can also be disposed immediately upstream of themirrors 25 to differently focus the light beams so that the sensors 18can be mounted directly to the mirror blocks 25 at equal distances.

Additionally, variable apertures could be used to vary the attributes,namely the depth of field and resolution of the captured images at agiven focal plane from one acquisition period to the next. From amanufacturing perspective, fixed apertures, and even variable apertures,can be less expensive and faster to position than variable liquidlenses.

The alternate optical configuration of FIG. 12 focuses incoming lightand passes the light beam through an aperture stop 13 and an aspheric orpositive lens 27 and a collimating or carrier lens 26. The light beam isthen split by beam splitter 19 and captured by sensors 18. Since eachsensor 18 is offset from the beam splitter at different distances, twofocal planes can be captured. The aperture stop 13 can also be avariable aperture stop and thus additional depth of field and resolutioninformation when varied over multiple acquisition periods.Alternatively, the aperture stop 13 can be the variably-polarizedaperture stop shown in FIG. 14 and further discussed below.

Digital image processing can combine each of the differently focused andseparately captured images by selecting and extracting the sharp areasof each image and combining them into a single full resolution image.Additionally, the color information from the blurred areas can bereconstructed using the contrast information of the sharp areas or thecombined image such that the colors are accurately reproduced.

First the fusion method generates a contrast weight map, a saturationweight map and an exposure weight map for each captured image. Thenthese maps are applied to select the best pixels from each image.Finally, the separate weighted images containing the selected orweighted pixels are combined with pyramid-based image fusion to generatea combined image.

By interpolating the color information, both resolution and contrast areslightly reduced. This, however, should not present a problem since theresolution of the sensors and combined image exceeds the resolution ofthe best endoscopes. On the other hand, the increased depth of focusallows for certain errors in the optics such as image field curvature tobe compensated. Image field curvature often occurs in endoscopes with avery long inversion system.

The extended camera head of FIG. 13 with a first segment includingelements 26 and 27, second segment 28 and third segment 29. The firstsegment includes an aperture 13, an aspheric lens 27 and a collimatinglens 26 as may the camera head in FIG. 12. The beam splitter 19 at theend of the first segment splits one third of the light onto a sensor 18and allows two-thirds of the light to pass through. Or, to compensatefor surface losses or other differences, the reflective surfaces insidethe beam splitters 19 may divide the beams unequally.

The second segment 28 is an inversion system carrying the remaininglight beam to a second beam splitter 19 which splits half or somefraction of the remaining light onto another sensor 18 in a differentfocal plane. The remaining one-third of the light beam passes throughthe third segment 29 which is an inversion system like that in thesecond segment 28. The remaining light is deflected by mirror 30 andimaged by sensor 18, which is also in a different focal plane. Eachinversion system flips the image or changes the parity of the imageresulting in various captured image orientations which must be correctedoptically or digitally.

The three sensors 18 in FIG. 13 are oriented such that all three can besoldered on the same board 72. In this way a very compact constructionis possible which can be accommodated in a video endoscope withoutchanging the housing or form factor of the device. Where appropriate,the deflecting mirror 30 can be replaced with another beam splitter 19to pass on some of the beam to an additional inversion system forfurther imaging, and so on. In this case, the reflectance of each beamsplitter 19 in the chain can be adjusted to equalize the light intensityimaged by each of the sensors 18.

The loss of light due to the distribution of the light beam onto varioussensors may be compensated in that the system can have a highernumerical aperture than an equivalent system, that is a system whichcovers the same depth of focus with a single sensor as this system doeswith multiple sensors.

With the higher numerical aperture, overall a higher resolution isachieved while in conventional systems this high resolution requires atrade-off of lower depth of field. Due to the fact that in the variousoptical arrangements above the same image is captured by various sensorsat the same time on different focal planes, the sharp areas of theindividual sensors can be combined into one image.

The camera head for an endoscope shown in FIG. 12 can further include aspecialized variably-polarized aperture stop 13 such as that shown inFIG. 14. This aperture stop 13 includes an outer opaque annular region31, an inner polarizing annular filter 32, and an innermost circularopening 33 with no filter. This graduated aperture provides differentf-numbers for beams of polarized and non-polarized light.

A beam exiting the aperture 13 of FIG. 14 is a separable beam includingtwo overlapping beams propagating together until divided by a polarizedbeam splitter. After separation, one beam has a higher f-number than theother beam. Thus, one beam can be imaged at high resolution and theother beam can be image at a high depth of field. A processing unit canthen calculate a single image with a higher depth of field and/or higherresolution than would be possible with a single sensor and conventionaloptical arrangements. The effect can be increased if the sensors 18 arein different focal planes. Alternatively, the polarizing filter 32 canbe a spectral filter. In addition, more levels or annular regions withinthe graduated aperture of FIG. 14 are also advantageous.

The outlined device in FIG. 15 shows an exemplary camera head with aspecialized Radio Frequency Identification (RFID) reader unit 36 fordetecting an RFID identifier (or tag) 38 for a specific endoscope type34. The camera head is surrounded by a handle structure 35 for easyhandling by the operator. The digital signals from the sensors are thensent via connecting line 37 to a processing unit or display unit.Alternatively, the digital processing of the sensor images can beperformed in the handle with only a combined image being sent to thedisplay.

Advantageously, one or more of the image sensors 18 can be connected toa small actuator 39 that can adjust the focal plane position. Thisallows the focal plane difference between the two sensors to be adjustedfor a particular situation without a variable liquid lens. The actuator39 can also be combined with these other modes to provide larger rangesof focal plane differences.

Upon the identification of the specific endoscope 34 from the tag 38 onthe proximal end of the endoscope, the actuator 39 adjusts the focalplanes of the sensors 18 to an optimal focal plane offset.Alternatively, the identification can be done via the camera head with aQR code, bar code, or a specific color scheme on the endoscope end.Additionally, the endoscope could be identified by direct connection viaa data bus or by analysis of electrical resistance or a magnetic fielddirection of the endoscope end.

The actuator 39 can be a piezo-electric motor or other small motor. Uponidentification of the endoscope tag 38, a RFID reader 36 of a camerahead like that in FIG. 12 signals a controller for the variable aperturestop 13. The variable aperture stop 13 is preferably disposed before thebeam splitter 19 for adjustment to an optimal focal plane offset.Alternatively, the controller could be linked to a variable liquid lens20 if the shown camera head was replaced with the camera head of FIG. 4.

It is also noted that any of the camera heads and optical arrangementsdisclosed herein may be implemented into the device of FIG. 15. Thecombined image from the several sensors of any of the camera heads willpreferably be calculated in real time for an image with increased depthof field and increased resolution. If this is not possible, then a realtime average value of the images super-imposed on each other can begenerated with the calculated image being available later. In addition,three-dimensional modeling from the different focal planes can becalculated either in real time and displayed on a three-dimensionaldisplay or calculated and generated later for analysis and diagnosis.

The invention being thus described, it will be obvious that the same maybe varied in many ways. For instance, capabilities, components orfeatures from each of the optical arrangements above are combinable ortransferrable to any of the other optical arrangements disclosed herein.Such variations are not to be regarded as a departure from the spiritand scope of the invention, and all such modifications as would beobvious to one skilled in the art are intended to be included within thescope of the following claims.

What is claimed is:
 1. An imaging head for an endoscope, comprising: afirst beamsplitter receiving a light beam from the endoscope, the firstbeamsplitter separating the light beam into at least a first portion anda second portion; a first aperture, having a first diameter, receivingthe first portion of light from the first beam splitter; a lensreceiving the first portion of light from the first aperture; a firstimage sensor receiving the first portion of light; a second aperture,having a second diameter, receiving the second portion of light, whereinthe first diameter of the first aperture is different from the seconddiameter of the second aperture; a second image sensor receiving thesecond portion of light; and an image processor connected to the firstand second image sensors, the imaging processor combining capturedimages from the first and second image sensors to produce a resultingimage with extended depth of field and/or improved resolution, whereinthe beamsplitter is not configured to split the light beam from theendoscope based on wavelength.
 2. The imaging head of claim 1, whereineither the first aperture or the second aperture or both the first andthe second apertures are variable apertures.
 3. The imaging head ofclaim 2, wherein the image processor is configured to combine capturedimages from several acquisition periods for the resulting image.
 4. Animaging head for an endoscope, comprising: a first beamsplitterreceiving a light beam from the endoscope, the first beamsplitterseparating the light beam into at least a first portion and a secondportion; a first aperture, having a first diameter, receiving the firstportion of light from the first beam splitter; a lens receiving thefirst portion of light from the first aperture; a first image sensorreceiving the first portion of light; a second aperture, having a seconddiameter, receiving the second portion of light, wherein the firstdiameter of the first aperture is different from the second diameter ofthe second aperture; a second image sensor receiving the second portionof light; a radio frequency identification (RFID) reader to detect anRFID identifier from the endoscope when the endoscope is attached to theimaging head; and an image processor connected to the first and secondimage sensors, the image processor combining captured images from thefirst and second image sensors to produce a resulting image withextended depth of field and/or improved resolution.
 5. The imaging headof claim 4 wherein the imaging processor identifies propertiesassociated with the attached endoscope based on the detected RFIDidentifier.
 6. The imaging head of claim 5, wherein the first apertureand the second aperture are variable apertures.
 7. The imaging head ofclaim 6, wherein captured images from several acquisition periods arecombined for the resulting image.